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ABSTRACT 

In  support  of  a  series  of  upcoming  Hydrocarbon  Boost  (HCB)  Tests,  facility  operation 
demonstrations  were  run  to  validate  the  Air  Force  Research  Laboratory  (AFRL)  Test  Stand  2A 
(TS2A)  Liquid  Oxygen  (LOX)  and  Gaseous  Nitrogen  (GN2)  propellant  systems  were  fully 
functional  and  able  to  meet  HCB  test  requirements.  While  facility  activation,  blowdown,  and 
system  checkout  tests  were  in  progress,  several  anomalies  were  detected.  First,  it  became 
apparent  after  a  few  tests,  substantial  audible  leaks  had  developed  in  two  control  valves  critical  to 
tank  pressurization.  Second,  anomalies  in  the  data  from  the  control  system  suggested  that  valve 
performance  had  changed  in  a  manner  suggesting  leakage.  These  two  issues  caused  TS2A  to 
suspend  operations  until  an  investigation  could  be  performed  to  determine  the  cause  of  the 
irregularities. 

The  respective  pressurization  and  vent  systems  were  partially  disassembled  in  order  to 
borescope  the  valves.  Subsequently,  it  was  discovered  there  was  significant  damage  to  the  valve 
plugs  and  seats.  The  valves  were  removed  and  replaced  with  valves  from  a  different 
manufacturer  that  could  meet  the  same  performance  characteristics.  The  replacement  valves 
were  previously  operated  in  similar  processes  and  had  proved  their  capability  to  function  under 
the  required  conditions.  Test  operations  were  restarted,  and  the  replacement  valves  were  closely 
monitored  to  verify  there  were  no  issues  with  their  operations;  there  were  none. 

This  paper  will  describe  the  nature  of  the  failure  of  the  valves,  and  the  impact  on  their 
capability  to  operate  effectively.  It  will  provide  a  failure  analysis  and  lessons  learned  so  that  future 
procurements  of  high-pressure,  high-performance  control  valves  can  avoid  the  mistakes  made  in 
the  design  and  procurement  of  these  kinds  of  valves. 

INTRODUCTION 

The  Experimental  Demonstration  Branch  at  the  AFRL  Aerospace  Systems  Directorate  at 
Edwards  AFB,  CA  began  system  activation  activities  of  the  TS2A  LOX  system  in  early  2012  to 
support  component  testing  for  the  HCB  program.  The  LOX  system  is  simple  and  comprised  of 
four  main  components:  GN2  pressurization,  pressurization  control  valves  (referred  to  as  XCVs), 
LOX  run  tank,  and  LOX  distribution  piping.  The  GN2  pressurization  system  and  XCVs  are  rated 
to  10,000  psig,  and  the  LOX  run  tank  to  8,500  psig.  After  activation  of  facility  mechanical,  control, 
and  instrumentation  systems,  we  began  performing  a  series  of  blowdown  tests  using  Liquid 
Nitrogen  (LN2)  for  system  shakedown  and  facility  performance  verification.  The  initial  test  was 
performed  using  500  psig  LOX  run  tank  pressurization,  with  subsequent  test  pressures  being 
increased  incrementally  up  to  the  required  test  pressure  of  6,000  psig.  Figure  1  shows  the 
pressurization  and  vent  systems  on  the  LOX  run  tank.  The  GN2  storage  cascade  is  not  shown, 
but  is  on  the  left  side  of  the  figure,  schematically,  with  the  flow  path  indicated  by  the  arrows. 
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Figure  1.  This  schematic  Process  and  Instrumentation  Diagram  (P&ID)  shows  how  the  LOX  run 
tank  is  pressurized  from  the  GN2  cascade,  as  well  as  the  system  used  to  vent  the  tank. 


During  initial  blowdown  tests,  we  started  to  notice  anomalies  in  the  tank  vent  rate  data 
that  made  us  suspect  the  vent  control  valve  (XCV-2010)  may  have  been  leaking.  The  leak  rate 
quickly  became  untenable  once  the  pressurization  system  reached  5,000  psig.  Not  only  was  the 
leak  now  obvious  in  the  data,  but  was  clearly  audible  out  on  the  test  stand.  Testing  was 
temporarily  suspended  in  order  to  fix  the  problem.  Leaking  control  valves  are  not  unusual, 
especially  large  metal-seated  globe  valves  operating  at  very  high  pressure.  We  have  a  great  deal 
of  experience  and  expertise  working  with  these  valves,  and  therefore  proceeded  with  our 
standard  “fix”  for  the  leak;  minor  mechanical  adjustments,  and  control  system  calibration  to 
ensure  that  the  plug  was  properly  seated  in  the  body  of  the  valve.  (Note  the  mechanical 
engineers  automatically  default  to  the  problem  being  a  control  system  problem,  and  vice-a-versa). 
These  fixes  are  generally  successful;  however,  in  this  case,  they  proved  to  be  futile.  The  leak 
continued  until  it  abruptly  increased  during  a  blowdown  test  whereupon  we  (the  mechanical 
engineers)  admitted  defeat  and  acknowledged  that  the  cause  could  only  be  attributed  to  a 
significant  mechanical  failure.  We  performed  a  borescopic  examination  and  discovered 
significant  damage  to  the  valve  plug  and  seat.  The  valve  was  removed,  examined,  and 
determined  to  be  severely  damaged  and  irreparable.  A  substitute  valve  was  installed  and 
activation  tests  resumed. 

Shortly  after  the  activation  tests  restarted,  we  noticed  that  the  tank  pressure  was  rising 
quickly  after  the  tank  vent  valve  (replacement  XCV-201 0)  was  closed.  This  was  noted  while 
performing  routine  leak  checks  on  the  main  tank  Pressure  Safety  Valve  (PSV).  This  pressure 
increase  was  significantly  higher  than  anything  we  had  seen  in  any  previous  operation.  The  only 
credible  source  of  the  gas  that  could  pressurize  the  tank  was  via  XCV-001 0.  (Note  that  XCV- 
001 1 ,  the  main  tank  pressurization  valve,  was  not  used  or  required  to  support  this  test  series.  It 
was  mechanically  locked  closed  and  previously  leak  checked).  Based  on  the  failure  of  XCV- 
2010,  we  (the  mechanical  engineers)  bypassed  blaming  the  controls  and  instrumentation  folks, 
and  proceeded  under  the  assumption  the  valve  had  mechanically  failed.  This  was  not  an 
unreasonable  assumption  for  three  reasons:  1 )  the  failure  was  much  more  abrupt  than  the  vent 
control  valve  failure,  2)  the  leak  rate  was  very  large,  and  3)  the  valve  was  from  the  same 
manufacturer  and  of  the  same  design  as  the  vent  valve.  Borescopic  examination  confirmed 
significant  damage  to  the  valve  plug  and  seat.  We  removed,  examined,  and  determined  the  valve 
to  be  severely  damaged  and  non-repairable.  A  substitute  valve  from  a  different  manufacturer 
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was  installed  and  activation  tests  resumed.  The  replacement  valve  has  a  proven  design,  had 
been  previously  used  in  high-pressure  systems,  and  had  an  identical  Cv  to  XCV-0010.  Some 
mechanical  modifications  were  required  to  accommodate  the  replacement  valve. 

FAILURE  ANALYSIS 

An  analysis  was  performed  to  determine  the  failure  mechanism  of  the  two  valves.  Since 
the  identical  failure  was  observed  in  two  previous  valves  of  the  same  design,  we  felt  it  was 
necessary  to  determine  the  failure  mode.  Three  other  valves  of  the  same  design  remained 
installed  in  the  system.  It  was  imperative  that  we  understood  how  XCV-0010  and  2010  failed  in 
order  to  evaluate  the  risk  of  leaving  the  remaining  three  valves  in  the  system. 

In  Figure  2,  we  show  the  basic  configuration  of  XCV-0010.  The  valve  is  a  hydraulically 
actuated  Y-pattern  globe  valve.  The  body  of  the  valve  is  constructed  from  304  stainless  steel  and 
the  plug  and  stem  are  made  from  A286  Gr  660  Type  2  stainless  steel.  The  plug  is  a  three-piece 
design  with  a  copper  seat. 


Figure  2.  A  cross  sectional  view  of  XCV-0010  shows  the  basic  configuration  of  the  valve.  This 
configuration,  consisting  of  a  Y-pattern  body,  plug  and  stem,  guides,  and  packing,  is  a  common 
configuration  for  high-pressure  globe  valves,  which  has  been  used  for  over  60  years. 

We  undertook  a  simple  three-phase  approach  to  determine  the  cause  of  the  valve 
failure(s):  1)  disassemble  the  valve  and  perform  a  postmortem  examination  to  evaluate  the  extent 
of  the  damage;  2)  using  the  postmortem  data,  determine  the  most  likely  mode  of  failure,  and 
develop  a  hypothesis  for  the  failure;  3)  build  a  model  of  the  valve  to  see  if  the  hypothesis  was 
supported  by  the  analysis. 
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POSTMORTEM  EXAMINATION 


Figures  3,  4,  and  5  show  the  extent  of  the  damage  to  the  valve. 


Removing  the  plug  from  the 
body  required  “tapping”  it  out. 
(Precision  valve  disassembly) 


The  guide  bearing  was  seized 
in  its  race  and  the  race  seized 
in  the  valve  body  (hence  the 
sledge  hammer) 


Figure  3.  Removing  the  plug  from  the  valve  body  was  very  difficult.  We  would  not  usually  use  a 
sledge  hammer  to  get  the  plug/stem  assembly  out  of  the  valve. 
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Copper  seating 
surface  extruded 


Extensive  plug  damage 
confirmed  borescope 
examination 


Severely  bent  ste 


Bronze  guide  bearing  material  deposited  on 
stem.  Bearing  stuck  In  the  race 


Figure  4.  Extensive  damage  to  the  plug  and  stem  was  found.  The  stainless  steel  on  the  plug  had 
been  scraped  off  in  the  area  where  the  plug  crashed  into  the  valve  body  seat.  The  copper  seat  on 
the  plug  was  badly  damaged  and  the  stem  was  severely  bent.  We  also  found  that  the  bronze 
guide  bearing  had  rubbed  so  hard  on  the  stem  once  it  bent,  that  the  bronze  was  deposited  onto 
the  stem  itself. 
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Copper  plug  material 
deposited  on  valve  seat 


Figure  5.  What  you  are  looking  at  is  the  seating  area  of  the  valve  body.  Note  that  stainless  steel 
in  the  seating  area  has  been  scrapped  so  hard  that  the  plug  contact  area  has  rolled  over  where 
the  plug  contacted.  Also,  you  can  see  where  we  found  copper  from  the  valve  plug  deposited  on 
the  valve  seat. 


FAILURE  HYPOTHESIS 

Upon  examination  of  the  disassembled  hardware,  the  failure  was  identified  as  a  bent 
stem.  At  this  point,  we  were  faced  with  determining  what  could  have  caused  the  stem  to  bend. 
There  are  only  two  modes  by  which  the  stem  could  have  bent  in  the  manner  it  did:  buckling 
and/or  a  cantilever  load  placing  a  bending  moment  on  the  stem.  Now,  intuitively  based  on  the 
evidence,  we  thought  that  the  stem  bending  could  only  have  occurred  due  to  a  bending  load; 
however,  we  have  clearly  demonstrated  on  other  valves  (much  to  our  embarrassment)  that  it  is 
entirely  possible  to  buckle  the  stem  by  over-adjusting  the  stem  projection  into  the  valve  seat  and 
then  applying  an  excessive  load  from  the  actuator.  In  the  past,  we  have  only  seen  this  happen  on 
very  small  valves  (less  than  %  inch),  where  the  I/d  ratio  is  high.  Unfortunately,  there  is  no  clear 
line  of  demarcation  in  regards  to  I/d  ratio  that  distinguishes  when  buckling  would  occur  that  can 
assist  us  in  making  this  determination.  However,  based  on  our  experience,  valves  sized  %  inch 
and  up  are  not  as  prone  to  stem  buckling.  Valves  the  size  of  XCV-0010  and  2010  are  highly 
unlikely  to  fail  in  this  manner.  Additionally,  the  bending  that  occurred  was  inconsistent  with  the 
manner  in  which  a  column  with  both  ends  fixed  would  bend,  lending  further  evidence  to  our  claim. 
Thus,  we  decided  to  start  our  analysis  by  building  a  model  in  SolidWorks  and  performing  a  finite 
element  stress  analysis.  A  cantilever  load  was  placed  on  the  stem  to  simulate  the  upstream 
pressure  experienced  during  valve  actuation. 

MODELING  AND  ANALYSIS 

Before  we  get  into  the  specifics  of  the  modeling  and  analysis,  it’s  necessary  to  describe 
how  the  valve  was  being  operated  when  it  failed.  Refer  to  Figure  1  as  necessary.  The  sequence 
of  events  is  as  follows: 
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1 .  Isolation  valves  (separating  the  GN2  supply  source  from  the  control  valves)  are 
opened  in  order  to  deliver  pressure  to  the  tank  pressure  control  valve  (XCV- 
0010).  At  this  point,  up  to  9,000  psig  GN2  is  on  the  upstream  side  of  XCV-001 0. 
The  downstream  side  of  the  tank  is  at  ambient  pressure,  so  the  pressure 
differential  on  the  valve  is  9,000  psi.  The  position  of  the  valve  is  such  that  the 
plug  and  stem  are  fixed  at  each  end  with  the  upper  support  being  the  guide 
bushing  and  actuator  and  the  lower  support  being  the  valve  seat. 

2.  The  vent  control  valve  (XCV-2010)  closes  in  preparation  for  the  LOX  run  tank 
pressurization.  The  tank  isolation  valve  (on  the  outlet  pipe)  is  already  closed,  as 
well  as  all  other  valves  connecting  the  tank  to  all  other  systems.  The  tank  is 
sealed  off  and  ready  to  be  pressurized. 

3.  The  control  system  commands  XCV-001 0  to  pressurize  the  tank  to  6,000  psig 
and  hold. 

4.  On  command,  XCV-001 0  goes  rapidly  to  the  full  open  position,  and  then  begins 
to  throttle  to  achieve  set  pressure  in  the  tank.  Once  target  pressure  is  achieved, 
the  valve  closes,  and  waits  for  the  test  to  start. 

5.  When  the  test  starts,  the  tank  isolation  valve  begins  to  open  again,  thus  XCV- 
001 0  experiences  a  pressure  drop  and  begins  to  throttle  open  to  maintain  the 
tank  pressure  at  the  set  point. 

6.  At  the  end  of  the  test,  XCV-001 0  closes  and  XCV-2010  (the  tank  vent  valve 
discussed  earlier)  begins  its  controlled  venting  of  the  LOX  run  tank. 

7.  Once  the  tank  is  vented,  all  the  valves  are  secured  to  their  normal  positions,  and 
the  system  is  returned  to  a  non-operating  secure  mode. 

In  step  4,  the  control  system  commands  XCV-001 0  to  go  full  open.  At  that  point,  the  plug 
comes  out  of  its  seat  under  the  9,000  psi  delta  pressure  and  the  valve  stem  assumes  the 
characteristics  of  a  cantilever  beam  (now  supported  at  only  one  end  by  the  guide  bushing).  This 
is  the  point  when  the  valve  stem  will  see  the  maximum  load  at  the  far  end  of  the  plug,  resulting  in 
the  maximum  bending  moment.  We  decided  to  start  our  analysis  using  these  conditions  as  we 
felt  that,  based  on  our  hypothesis,  this  would  be  the  worst-case  load  on  the  stem. 

We  started  with  a  “first  order”  conservative  analysis.  By  “first  order”  conservative 
analysis,  we  mean  that  we  modeled  the  valve  to  be  “better”  than  it  really  was.  The  following 
constraints/assumptions  used  in  the  analysis  will  explain: 

1 .  We  did  not  have  complete  engineering  drawings  for  the  valves  that  failed.  Thus, 
we  modeled  the  stem  and  plug  geometry  from  the  failed  hardware.  We  had  to 
build  our  own  solid  models  of  the  hardware  as  best  we  could  from  the  existing 
parts,  and  by  scaling  from  the  assembly  drawing. 

2.  We  used  the  9,000  psi  differential  pressure  as  the  load  force  since  we  had  test 
data  to  support  that. 

3.  We  assumed  that  the  guide  bearing  was  in  intimate  contact  with  the  valve  stem 
(which  it  was  not;  it  was  a  rather  sloppy  fit),  and  the  guide  bearing  was  somewhat 
closer  to  the  plug  than  it  really  was.  It  should  be  noted  that  the  first  cut  of  the 
FEA  had  to  be  modified  to  move  the  support  bearing  closer  to  the  plug  because 
the  large  displacement  would  not  allow  the  model  to  converge.  This  was  a 
conservative  assumption  made  because  we  didn’t  have  the  fabrication  drawings. 

4.  We  assumed  the  bending  force  was  acting  on  the  plug  only.  This  reduced  the 
area  of  the  application  of  the  pressure,  resulting  in  a  smaller  total  force  on  the 
stem. 

5.  We  used  the  high  end  of  the  material  properties  for  A286  Gr  660  Type  2  stainless 
steel.  (171  ksi  Su,  112  ksi  Sy) 
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We  constructed  the  models  based  on  these  constraints  and  assumptions  in  SolidWorks, 
and  performed  a  series  of  finite  element  analysis  studies.  FEA  studies  were  performed  for  both 
bending  and  buckling.  The  valve  failed  in  bending  by  a  large  margin  consistent  with  the 
postmortem  examination  of  the  valve.  The  analysis  indicated  that  the  factor  of  safety  (FOS)  in 
buckling  was  8  (which  means,  the  load  required  to  make  the  stem  fail  is  eight  times  larger  than 
the  axial  load  applied  to  the  stem  by  the  actuator  in  the  closed  valve  position),  so  we  decided  that 
no  further  buckling  analysis  was  warranted.  The  results  are  presented  in  and  discussed  with 
Figures,  6,  7,  and  8. 


Figure  6.  This  graphic  shows  the  output  of  the  finite  element  solver  displaying  the  maximum  von 
Mises  stress  for  the  XCV-0010  stem/plug.  Notice,  as  we  stated  in  our  assumptions,  that  the  force 
of  the  pressure  was  placed  orthogonal  to  the  face  of  the  plug,  but  to  be  conservative,  only  applied 
to  the  projection  of  the  plug  tip  and  not  the  whole  wetted  area  of  the  plug  (the  large  diameter 
portion).  The  results  show  that  the  maximum  stress  on  the  stem  (487  ksi)  occurs  at  the  location 
of  the  cantilever  support  (guide  bearing),  and  greatly  exceeds  the  yield  strength  of  the  stem, 
resulting  in  permanent  deformation.  If  you  look  carefully  at  the  centerline,  you  will  also  notice 
slight  bending  of  the  stem.  This  is  less  than  the  real  stem,  but  shows  that  even  with  this  very 
conservative  analysis,  the  stem  would  bend.  If  the  maximum  stress  had  not  exceeded  the 
material  allowable,  you  would  only  see  displacement  on  a  greatly  exaggerated  (50x  or  greater) 
graphic  (see  next  figure). 
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Figure  7.  Seen  here  is  the  displacement  exaggerated  by  10  times  to  show  the  bending  of  the 
stem  beginning  at  the  location  of  the  lower  guide  bearing.  The  maximum  displacement  according 
to  the  conservative  model  is  approximately  0.09  inches.  It  doesn’t  seem  like  much,  but 
permanent  deformation  of  the  stem  on  this  magnitude  indicates  both  an  extreme  overload,  and  in 
this  case,  certain  interference  between  the  plug  and  the  valve  seat. 


Figure  8.  Here  you  can  see  the  results  of  the  finite  element  solver  displaying  the  FOS  for  the 
XCV-001 0  stem/plug.  The  results  show  that  the  FOS  was  less  than  one  (1 )  in  many  areas  of  the 
structure  with  the  minimum  being  0.21.  This  means  that  permanent  deformation  occurred.  No 
displacement  is  shown  in  this  particular  graphic. 
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For  the  sake  of  completeness  an  FEA  was  performed  on  the  replacement  valve  (XCV- 
3003)  that  was  taken  from  the  test  stand  hydrogen  system.  The  same  analysis  assumptions 
were  made  for  this  valve  with  two  twists:  First,  we  projected  the  pressure  force  on  the  full 
projection  of  the  plug  (as  seen  by  the  fluid  flow),  and  second  we  used  the  low  end  of  the  material 
properties  for  the  Nitronic  50  stainless  steel  stem  in  this  valve  (135  ksi  Su,  and  105  ksi  Sy).  We 
did  this  to  see  if  the  valve  would  pass  using  a  similar  analysis  technique,  but  with  much  less 
conservatism.  The  results  are  presented  in  and  discussed  with  Figures,  9, 10,  and  11. 


Figure  9.  This  graphic  shows  the  output  of  the  finite  element  solver  displaying  the  maximum  von 
Mises  stress  for  the  XCV-3003  stem/plug.  Notice,  as  we  stated  in  our  assumptions,  that  the  force 
of  the  pressure  was  placed  orthogonal  to  the  face  of  the  plug,  however  in  this  less  conservative 
case,  applied  to  the  whole  projection  of  the  plug  tip.  The  results  show  that  the  maximum  stress 
on  the  stem  occurs  at  the  location  of  the  cantilever  support  (guide  bearing),  but  in  the  case  of  this 
valve,  the  stress  does  not  exceed  the  yield  strength  of  the  material.  Note  that  the  deflection  in 
this  graphic  is  exaggerated  by  100  times.  Otherwise,  as  in  Figure  6,  it  is  undetectable. 
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Figure  10.  Seen  here  is  the  displacement  results  showing  the  maximum  displacement  is  at  least 
an  order  of  magnitude  less  than  XCV-001 0,  and  would  return  to  zero  once  the  load  is  removed 
since  no  permanent  yielding  took  place. 

All  of  the  work  (disassembly,  postmortem  inspection,  analysis,  etc.)  was  performed  for 
the  tank  vent  control  valve  (XCV-201 0).  The  failure  was  identical  to  that  of  XCV-001 1 ,  and  the 
analysis  showed  similar  results,  so  this  paper  will  not  discuss  this  particular  valve  any  further. 
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Figure  11.  Here  you  can  see  the  results  of  the  finite  element  solver  displaying  the  FOS  for  the 
XCV-3003  stem/plug.  The  results  show  that  the  FOS  was  greater  than  one  (1)  in  all  areas  of  the 
structure  with  the  minimum  being  1 .60.  This  means  that  the  FEA  indicated  no  permanent  yielding 
of  the  stem. 

FAILURE  CAUSE,  PREVENTION,  CORRECTIVE  ACTIONS,  AND  RECOMMENDATIONS 

The  cause  of  the  failure  appears  to  be  purely  mechanical  as  there  were  no  control 
system  or  operation  errors  that  could  be  traced  to  the  valve  failures.  Several  obvious  design 
flaws  in  the  valve  are  considered  to  be  major  contributors  to  the  failure  of  the  valve.  First,  the 
large  surface  area  projection  of  the  plug  into  the  flow  stream  subjects  the  stem  and  plug  to  an 
excessive  load.  Second,  the  distance  from  the  stem  guide  bearing  to  the  plug  creates  a  long 
cantilever  arm  that  promotes  bending  of  the  stem.  This  feature  is  not  typical  of  other  high 
pressure  valves,  which  have  guide  bearings  located  much  closer  to  the  plug  and  have  proven 
reliable  in  similar/identical  environments.  Third,  the  stem  guide  bearing  tolerance  was  such  that 
the  stem  was  not  well  supported  by  the  bearing,  thus  lending  opportunity  to  produce  greater 
displacements  and  higher  stress  concentrations  on  the  valve  stem.  Finally,  the  ratio  of  the  stem 
diameter  to  the  plug  diameter  (dstem/dpiug)  is  low.  This  feature  creates  a  higher  stress 
concentration  at  the  stem-plug  interface. 

To  prevent  this  from  occurring  again  in  the  as-built  system,  we  found  the  location  of  all 
the  installed  valves  that  have  a  similar  design  by  the  same  manufacturer.  There  were  two 
additional  valves  identified  on  the  installed  system,  and  one  installed  on  the  test  skid  for  the  HCB 
program.  We  took  the  following  steps  to  determine  the  extent  of  the  problem  and  mitigate  the 
program  impact: 

1 .  We  verified  the  service  requirements  for  each  valve  in  order  to  determine 
whether  they  were  susceptible  to  a  similar  type  of  failure. 
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2.  We  removed  and  replaced  “high  risk”  valves  where  possible  and  tagged  those 
valves  to  identify  that  they  are  not  approved  for  future  use  in  similar  high- 
pressure  applications. 

3.  We  located  out-of-service  valves  of  the  same  design  and  identified  them  for  non¬ 
use  pending  an  engineering  evaluation  for  proposed  future  service. 

The  four  high-pressure  valves  in  the  LOX  facility  that  are  of  the  same  design,  are  the  two 
previously  discussed  as  well  as  two  others  that  remain  installed:  the  LOX  run  tank  fill  isolation 
valve,  and  the  LOX  run  tank  emergency  vent  and  overfill  valve.  The  LOX  run  tank  fill  isolation 
valve  was  determined  to  be  very  low  risk  of  the  same  failure  because  this  valve  would  never  be 
opened  against  more  than  about  150  psi  differential  pressure.  While  seated,  the  valve  should 
see  no  undue  stress  even  when  the  tank  is  pressurized  to  8,500  psig.  No  leaks  or  abnormal 
operations  have  been  noted  with  this  valve. 

The  LOX  run  tank  emergency  vent  valve,  on  the  other  hand,  is  a  whole  different  story. 
This  valve  is  designed  to  be  actuated,  either  manually  or  automatically,  in  the  event  of  tank  over¬ 
pressurization.  This  would  potentially  subject  the  valve  to  an  8,500  psi  delta  pressure.  This  valve 
is  considered  by  us  (TS2A)  to  have  a  high  risk  of  failure  in  the  event  of  an  emergency  tank  vent. 
The  valve  would  probably  work  one  time,  but  may  not  be  functional  after  a  single  use.  There  is 
also  a  possibility  that  the  valve  would  not  reseat  after  opening  due  to  permanent  material 
deformation,  caused  by  the  large  pressure  differential,  thus  potentially  allowing  contamination  into 
the  clean  system  from  atmosphere  -  an  extremely  costly  impact.  The  short  term  solution  for 
dealing  with  this  situation  is,  first,  to  do  everything  possible  to  prevent  an  emergency  vent 
situation.  Naturally,  this  is  something  we  would  always  do,  but  because  of  the  critical  nature  of 
this  problem  we  are  looking  into  adding  backup  controls  and  red-line  cuts  to  the  pressurization 
using  redundant  instrumentation.  An  intermediate  solution  (currently  in  design)  is  to  install  a 
temporary  replacement  at  the  conclusion  of  the  current  test  campaign.  This  would  allow  for 
proceeding  into  the  next  phase  of  testing,  but  would  limit  run  tank  operations  to  about  6,000  psig. 
The  long  term  solution  we  are  pursuing  is  a  brand  new  replacement  valve;  however,  a  4”  8,500 
psig  rated  Monel  globe  valve  does  not  come  cheap  (several  hundred  thousand  dollars)  and  would 
take  a  substantial  amount  of  time  to  procure. 

LESSONS  LEARNED 

There  were  several  lessons  we  learned  from  the  failure  and  subsequent  analysis  of  these 
two  control  valves: 

1 .  For  initial  testing  of  high-pressure  systems,  stop  at  incremental  points  on  the  way 
to  MAWP  and  perform  extensive  system  inspections.  Doing  so  will  afford 
operators,  engineers  and  maintainers  an  opportunity  to  closely  examine  system 
integrity  and  performance  and  identify/correct  any  potential  concerns  prior  to 
experiencing  system  failure.  This  practice  could  also  potentially  save  a  great 
deal  of  time,  money  and  effort  spent  on  your  program. 

2.  If  using  a  new  or  unproven  supplier  or  design: 

•  Insist  on  more  analysis,  testing,  and  design  reviews  to  prove  the  design 
is  adequate  to  meet  the  requirements. 

•  Have  the  supplier  available  on  site  for  inspection  of  the  equipment 
installed  in  the  system. 

•  Talk  to  other  supplier  customers  to  get  their  input  on  the  performance  of 
similar  equipment. 

•  Spend  more  time  performing  critical  evaluation  of  the  design  in  order  to 
identify  potential  design  flaws  prior  to  implementing  new  equipment. 

3.  Don’t  get  target  fixation  (concentrate  only  on  one  failure  mode): 
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•  We  thought  it  was  a  controls/tuning/calibration/adjustment  issue  because 
we  have  seen  those  many  times. 

•  Didn’t  consider  mechanical  failure  because  we’d  never  seen  something 
like  this  before  and  never  imagined  it  could  happen. 

4.  Perform  more  inspections  of  the  valves  between  tests  to  monitor  their  condition 
until  we  have  confidence  they  are  operating  within  normal  parameters. 

5.  Make  absolutely  sure  the  contractor  understands  the  requirements: 

•  Perform  a  more  thorough  prequalification  review. 

•  Review  operating  scenarios  (not  just  the  specifications)  with  the 
contractor. 

•  Ask  them  lots  of  questions  to  verify  they  understand  your  requirements. 

•  Don’t  assume  the  contractor  understands  your  system  operating 
procedures. 


CONCLUSION 

Poor  valve  design  was  the  single  largest  contributor  to  the  failure.  There  are  systemic 
design  problems  with  this  particular  valve  configuration  that  were  not  detected  during 
procurement.  We  had  no  experience  with  this  valve  brand  and  design,  but  the  contract  was 
awarded  to  the  vendor  due  to  their  meeting  the  Scope  of  Work  (SOW)  qualifications,  and 
because  they  were  the  low  bidder.  The  issue  was  not  detected  until  about  10  years  after 
installation  of  the  valves  because  we  had  never  operated  them  at  conditions  anywhere  near 
9,000  psig.  Previous  experience  with  similarly  designed  valves  from  another  supplier  gave  us  no 
indication  that  we  would  have  this  sort  of  failure.  Communication  issues  with  the  supplier  on 
exactly  how  we  operate  the  valves  were  also  a  contributing  factor  in  the  failures.  The  main 
supplier  of  our  high  pressure  control  valves  (for  the  last  25  years)  is  intimately  familiar  with  how 
we  operate  them  in  a  high-pressure  component  test  stand  like  TS2A.  It  was  assumed  that  the 
supplier  of  these  valves  had  an  equal  understanding  of  our  operations.  This  was  a  bad 
assumption,  which  resulted  in  a  great  deal  of  time,  money  and  effort  being  spent  to  correct  what 
could  have  easily  been  prevented. 
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